Performance Evaluation and Benchmarking for the Era of Artificial Intelligence by Unknown

Author:Unknown
Language: eng
Format: epub
ISBN: 9783030114046
Publisher: Springer International Publishing

The iterative stage of model training and optimization of the hyper-parameters (parameters of the chosen models and the training algorithm) until reaching the desired model accuracy. This iterative cycle often goes back to the preparation stage if the dataset is incomplete or has flaws or if the engineering features or model choice must be reviewed or the technical environment of the training execution has reached a hard limit that blocks the whole process.

The preparation phase depends very strongly on human factors. The data preparation stage is one of the critical steps of the process and is entirely under human supervision (often subject matter experts) and their total automation is almost impossible. However, some sub-steps can be automated. One can cite for example: the data-augmentation, the data denoising, the data formatting/reformatting, the signal pre-processing and the data fusion (Compute vision, Acoustic, Lidar point clouds), the data semi-auto labeling. The duration of the cycle of training and optimization of the hyper-parameters is dominated by two factors: the training time of the models which is by far the heavier and longer machine time and the faculty to quickly find the values of the hyper-parameters which maximizes the model accuracy. The technical preparation stage also depends on the context, plus human and organization choices. The optimization of hyper-parameters is often performed by experienced data scientists and there exist optimum search techniques and tools that automate this task.

Inference Phase

The inference phase depends first and foremost on the targeted deployment platform: this concerns data center servers or on the edge devices (embedded systems, smart camera, smart phone, embedded card in vehicles or robots). Their platform characteristics are primarily for the Inference Performance: presence or not of hardware AI Accelerators, RAM capacity, CPU, IO, OS, framework, types of sensors, process performances and energy consumption, etc.). These factors strongly condition the parameters of the stage of model packaging to adapt them to the targeted deployment platform. This could include the model reduction to adapt to the platform hardware footprint (memory size, compute capacity, energy consumption), response time performance improvement, the integration of data pre-processing and post-processing logics, and some extra logical processing workflow (including calling other models) and then packaging in a secured format compatible with the target deployment platform. Finally comes the model deployment followed by the solution run-time and exploitation with the objective of achieving the desired performance in the conditions and constraints of actual operation in terms of model prediction accuracy and end-to-end processing time.

Another important stress factor to consider for the Training and Inference platforms is related to the level of workload concurrency introduced by parallel incoming requests. This could be illustrated for the Training phase, when several users share the same training platform and may concurrently submit intensive Training tasks. For the Inference phase, it is also very likely that several incoming detection requests have to be proceed by a platform at the same time.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.